Relationships between Gaussian processes, Support Vector machines and Smoothing Splines

Bayesian Gaussian processes and Support Vector machines are powerful kernel-based methods to attack the pattern recognition problem. Probably due to the very different philosophies of the fields they have been originally proposed in, techniques for these two models have been developed somewhat in isolation from each other. This tutorial paper reviews relationships between Bayesian Gaussian processes and Support Vector machines. We show how in a certain welldefined sense both emerge as special cases of smoothing Spline models, and our principal aim is to develop this correspondence as clearly and simply as possible. We also clarify notions of feature space and include essential material on Reproducing Kernel Hilbert spaces as the basic hypothesis spaces of smoothing Spline and other kernel methods.